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ABSTRACT 

Scholarly communication is in a perpetual state of disruption. Within this, peer review of research articles remains an 
essential part of the formal publication process, distinguishing it from virtually all other modes of communication. In the 
last several years, there has been an explosive wave of innovation in peer review research, platforms, discussions, tools and 
services. This is largely coupled with the ongoing and parallel evolution of scholarly communication as it adapts to rapidly 
changing environments, within what is widely considered as the ‘open research’ or ‘open science’ movement. Here, we 
summarise the current ebb and flow around changes to peer review and consider its role in a modern digital research and 
communications infrastructure and suggest why uptake of new models of peer review appears to have been so low 
compared to what is often viewed as the ‘traditional’ method of peer review. Finally, we offer some insight into the 
potential futures of scholarly peer review and consider what impacts this might have on the broader scholarly research 
ecosystem. In particular, we focus on the key traits of certification and reputation, moderation and quality control and 
engagement incentives, and discuss how these interact with socio-technical aspects of peer review and academic culture. 
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INTRODUCTION 

Peer review is one of the strongest social constructs within the 
self-regulated world of academia and scholarly communication. 
Researcher attitudes towards peer review are often in a state 
of reverence hailing it as a ‘golden standard’ (Mayden 2012; 
D’Andrea, James and O’Dwyer 2017), and even sometimes the 
distinction between a binary state of ‘verified’ and ‘unverified’ 
for research papers published in scholarly journals. Having a 
piece of research, including articles, books, conference proceed¬ 
ings and even grant applications, attain the status of 'peer re¬ 
viewed’ is considered to be a defining moment in the career of 
any scholar, and it has an incredible amount of capital attributed 
to it for scholarly reputation. Peer review is purported to serve 
many functions, including quality control as a screening mecha¬ 
nism, legitimisation of scientific research and the self-regulation 
of scientific communities. As such, in modern academia peer re¬ 
view remains critical in defining professional advancement and 


the hierarchical structure of research institutes (Fyfe et al. 2017; 
Moore et al. 2017), and is generally held in high regard across re¬ 
search communities (Goodman 1994; Bedeian 2003; Ware, 2011, 
2015; Pierson 2018; Jutta andFredrik 2016). 

With so much standing attributed to peer review, one would 
expect that it is a relatively optimised process, generally well- 
understood as a theory and a practice, and stable due to its 
widespread adoption and acceptance as a method. However, the 
reality could not be further from the truth. Since its origins, 
there have been vocal critics about almost every aspect and form 
of peer review, from its implementation and management, to 
its wider effects on research culture and the dissemination of 
scholarship (SI 1) (Smith 2006). However, widespread acceptance 
of these criticisms, and a desire to improve peer review based on 
them, can lead to a state of cognitive dissonance for researchers, 
as criticisms can be interpreted as undermining or challenging 
the foundations of scholarship itself, as well as the legitimacy 
of research communities. On the other hand, it is also often 
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considered to be the ‘best that we have’, with scholars remain¬ 
ing frustrated but with the view that alternatives to peer review 
are always less optimal; irrespective of the diversity of differ¬ 
ent forms of peer review (Jubb 2016). Thus, scholarly research is 
in a state of begrudging acceptance of the present state of peer 
review, despite decades of criticisms and little evidence that it 
even fulfils the process it is purported to do (for summaries of 
these criticisms, see Walker and Rocha da Silva 2015; Tennant 
et al. 2017; Ross-Hellauer 2017a). 

What is currently clear is that there is a divergence between 
how peer review is generally practiced as a multi-dimensional 
and diverse suite of processes, and how it is commonly regarded 
as a singular ideologue (Pontille and Torny 2015; Ross-Hellauer 
2017b; Casnici Niccolo et al. 2017), despite substantial evidence 
for a wide range of inherent biases (Lee et al. 2013; Helmer et al. 
2017; Kuehn 2017; Tennant 2017; Tomkins, Zhang and Heavlin 
2017; Iezzoni 2018). It is now not uncommon to hear views about 
how to optimise peer review based on smaller population-level 
studies (e.g. journal- or discipline-level), with polar opposite pro¬ 
cesses conflictingly hailed as solutions to the same underlying 
problems (e.g. reviewer blinding versus identification) (Bastian 
2017; Tennant 2017). Such diversity of views, while welcomed, 
has generally inhibited the development of any sort of stan¬ 
dardisation behind the definition or the process (Ross-Hellauer 
2017a; Allen et al. 2018), which, in turn, has created a fragmented 
landscape and makes any sort of comparative assessment into 
the efficacy of peer review problematic (Squazzoni, Grimaldo 
and Marusic 2017). This has been exacerbated by a general lack 
of coherence in the implementation of peer review, as well as a 
paucity of evidence and data sharing on it, making it difficult to 
draw rigorous conclusions about peer review systems. This con¬ 
flation of the ideal with the process can also be extremely dam¬ 
aging, as it creates a divergence where trust is expected to fill the 
gaps, and therefore can be used to undermine the credibility of 
scholarly research. Thus, the historical controversies surround¬ 
ing peer review are still reflected in its present systemic state 
(Thomas 2018). 

While the origins of peer-to-peer evaluation can be traced 
back to the very origins of scholarship (Kronick 1990; Spier 2002; 
Csiszar 2016; Melinda 2017; Moxham and Fyfe 2017), the advent 
of what many regard now as the editor-led ‘traditional’ process 
of ‘peer review’ is only relatively recent, having been established 
in a piecemeal fashion during the middle of the 20 th Century 
(Zuckerman and Merton 1971; Baldwin 2015). The formalised 
practice itself only began a century or so before, as part of a 
community-governed process associated with learned societies 
and their early scholarly journals (Moxham and Fyfe 2017). Here, 
contrary to how many academics often view the modern pro¬ 
cess (Nicholas et al. 2015), peer review was employed mostly to 
help constructively improve manuscripts by eliminating obvious 
flaws and gaps in reasoning and improving the rhetorical style 
and argumentation of articles, rather for any sort of implicit or 
explicit gatekeeping function. A key here was that peer reviewed 
scholarly journals became a way of providing scientific legiti¬ 
macy to learned societies, which was reciprocated by those so¬ 
cieties through providing authoritative credibility to those jour¬ 
nals. Institutionalisation of the review process took place during 
the 20 th Century, in order to help handle problems with the num¬ 
ber of research articles being submitted (i.e. as a gatekeeping or 
filtering process), as well as to meet increasing demands for ex¬ 
pert authority in a research world that was becoming rapidly 
specialised. One consequence of this was the synonymisation 
of peer review with scholarly value, which catalysed commer¬ 
cial interest in the process, as it became a way of strengthen¬ 


ing journal brands for marketing purposes. An additional ef¬ 
fect of this was the effective out-sourcing of the governance of 
peer review, and the legitimisation that came with it, to com¬ 
mercial entities that operated outside of research communities. 
Recent developments in ‘Open Peer Review’ (OPR) can largely 
be viewed as a set of practices to streamline and improve the 
process in a variety of ways, to help realign the modem prac¬ 
tices with the original ideals of progressive collaboration and im¬ 
proving the argumentation style (Melinda 2017; Ross-Hellauer, 
Deppe and Schmidt 2017; Ross-Hellauer 2017a; Ross-Hellauer 
2017b), and to return peer review to its collegial, constructive 
origins. 

Despite the critical importance of peer review in scholarly 
communication, and considerable recent effort to understand 
and improve the process, there remain numerous key issues. 
Some of the main ones include: 

• A lack of adequate training and support for researchers in 
best practices for how to perform peer review (or respond to 
peer reviews) (Schroter et al. 2004); 

• The length of time taken for the peer review process (Born- 
mann and Daniel 2010; Lyman 2013); 

• That valuable contextual information is often lost as review 
reports remain unpublished (Walker and Rocha da Silva 2015; 
Ross-Hellauer 2017a); 

• What the best operational processes should be for different 
research communities (Bruce et al. 2016); 

• A general lack of rigorous evidence into the functionality of 
different elements of peer review, including quality (Lee and 
Moher 2017; Squazzoni, Brezis and Marusic 2017; Squazzoni, 
Grimaldo and Marusic 2017); 

• The relationship between peer review quality and journal 
quality (Pierson 2018); 

• Core competences and standards for editors engaged in peer 
review (Moher et al. 2017). 

• Any form of strategy or consensus on how to address some of 
the major criticisms levied at peer review (Walker and Rocha 
da Silva 2015; Tennant et al. 2017; Thomas 2018). 

In spite of these challenges, considerable progress in under¬ 
standing of peer review has been made in recent years and help¬ 
ing to fill in our knowledge gaps about the process. Alongside 
this, wealth of new platforms and services have emerged that 
are attempting to resolve some of the socio-technological is¬ 
sues associated with peer review, which has been termed the 
‘peer review revolution’ (J. P. Tennant et al. 2017). There has also 
been an emergence of new interest and data gathering, helping 
to ignite a new wave of cross-stakeholder discussions and re¬ 
search in to the theory and practice of peer review (Ware 2011; 
Kovanis et al. 2017; Squazzoni, Brezis and Marusic 2017; Sizo, 
Lino and Rocha 2018). These ongoing developments are critical 
to the future of scholarly research, its communication, and the 
foundational structures of scholarly communities around 
the world. The purpose of this article is to summarise some of 
the key elements of the present state of peer review, and hope¬ 
fully catalyse wider critical discussions and more diverse inno¬ 
vations for its future. 

THE PRESENT STATE OF SCHOLARLY PEER 
REVIEW 

It is estimated that more than 2.5 million English language sci¬ 
entific research publications are now published each year and at 
a rapidly increasing rate (Ware and Mabe 2015). This creates an 
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incredible burden on the global research workforce, considering 
that a typical research paper requires 2-3 referees and a han¬ 
dling editor, most of whom act on a volunteer basis for scholarly 
journals. This has created a state commonly referred to as 're¬ 
viewer fatigue’ (Breuning et al. 2015; C. W. Fox, Albert and Vines 
2017), and available evidence suggests that the majority of re¬ 
views are performed by a minority of researchers within an in¬ 
creasingly over-burdened system (Lyman 2013; Jubb 2016; Gropp 
et al. 2017; Kovanis et al. 2017; J. Fox and Petchey 2010; Vines, 
Rieseberg and Smith 2010). This burden also appears to be un¬ 
evenly distributed geographically, with Chinese authors review¬ 
ing proportionally less articles than western authors (Jubb 2016). 
Several solutions have been proposed to resolve this state, in¬ 
cluding how to incentivise more researchers to engage with the 
review process, with a focus here primarily on quantity rather 
than quality. This has been directly tied with developments in 
how to appropriately accredit effort from peer reviewers, such 
as how to include this work in hiring, promotion, and tenure 
processes, in which peer review is typically almost entirely 
absent. 

These dual issues of incentivisation and reputation or 
certification are coupled with a third major issue, effective 
moderation, which is typically an opaque editorial-controlled 
function with little standardisation across journals (Moher et al. 
2017), but seen as being crucial for injecting any sort of verifi¬ 
cation or validity into the review process. This aspect is criti¬ 
cal for peer review, as typically moderated peer review is seen 
as the process that differentiates it from other forms of grey 
literature. 

These three factors (incentivisation for engagement, certifi¬ 
cation and reputation and moderation as a quality control pro¬ 
cess) are unified by the more complex issue of transparency in 
peer review, which itself is part of wider changes in the schol¬ 
arly communication system around the advent of 'open science’. 
While there is no single, accepted, unified definition or vision 
of 'open science’, one of the core aspects of it revolves around 
greater transparency throughout the entire research process, in¬ 
cluding peer review. There are numerous reasons often given for 
this, such as to combat the 'reproducibility crisis’, to expose or 
prevent research misconduct, to introduce greater accountabil¬ 
ity for researchers, or to increase the verifiability of the research 
record in order to engender greater public trust for the scientific 
enterprise (Morey et al. 2016). However, at the present there re¬ 
mains little consensus on the optimal way in which to resolve 
any of these issues, despite an increasing interest and dedicated 
research into them. 

Much of the current research into peer review focuses on 
the functionality of the traditional process, its performance and 
the dimensions of bias (Lee and Moher 2017; Squazzoni, Brezis 
and Marusic 2017; Squazzoni, Grimaldo and Marusic 2017). As 
such, improvements to peer review are often centred around 
these, and are very journal-centric or article-centric by nature— 
primarily because these are the principle data source. One re¬ 
sult of this is that many of the supposed innovations are fairly 
limited in scope, within the diverse realm of scholarly com¬ 
munication, and tend to be focused within this framework. 
Consequently, they do little to address the wider issues re¬ 
lated to scholarly journals (e.g. journal ranking in research as¬ 
sessment) and articles (e.g. appropriate accreditation) as prin¬ 
ciple forms of scholarly communication (Brembs, Button, and 
Munafo 2013). 

Despite this generally narrow field of view, there have 
been numerous recent suggestions about entirely novel meth¬ 
ods for scholarly communication and peer review, which have 


the potential to help solve many of these issues (Priem and 
Hemminger 2012; Wellen 2013; Nwagwu and Onyancha 2015; 
Tennant et al. 2017; Schmidt and Gorogh 2017; Heller and 
Bartling 2014). However, one of the consequences of the way 
the current peer review system operates is that of cultural in¬ 
ertia (Jonasson 2016), or at least slow rates of adoption, which 
largely remain in spite of any changes to the surrounding envi¬ 
ronment. To demonstrate that any new service or platform op¬ 
erates more effectively than current processes, those services 
must be able to empirically show this in order to obtain any 
sort of sustainable user base (note, here we mean sustainabil¬ 
ity for the long-term operation of the platform). However, those 
same services cannot in turn acquire appropriate usage data for 
this, as they struggle to acquire the users they need to effectively 
demonstrate an optimised alternative process needed to incen¬ 
tivise engagement. One key issue here is that the value of peer 
review as academic capital is often concealed or very difficult to 
measure, which makes development of incentives to adopt in¬ 
novative models and practices problematic. The consequence of 
this is that it all creates a cycle of inertia, where innovations and 
adoption of those innovations remains fairly stagnant relative to 
the sustained use of more familiar journal-coupled processes, 
and progress towards any optimised system remains slow. Such 
a psychological phenomenon is known as the ‘penguin effect’, 
whereby a physiological crowd mentality suppresses any experi¬ 
mentation beyond that crowd due to the perception of increased 
risk and lack of incentive to change (Choi 1994). For now, in 2018, 
we remain with a scholarly communication system based on a 
19 th Century process of peer review embedded into a 17 th Cen¬ 
tury method of communication. 


INNOVATIONS IN ‘OPEN PEER REVIEW’ 

Due to the intrinsic coupling between peer review and scholarly 
journals, disruptions in peer review are part of a much wider 
paradigm shift in scholarly communication. Both traditional and 
newer service vendors are experimenting with a wider range 
of new models, regarded as a ‘revolutionary phase’ in peer re¬ 
view (Tennant et al. 2017). This has come from a combina¬ 
tion of actors, including learned societies and a range of for- 
profit and non-profit entities, which raises questions around 
governance structures within scholarly communication and 
peer review due to the inherent legitimacy associated with the 
process. One such example is that around responsibility and ac¬ 
countability in peer review, created by the different relationships 
that exist between researchers and learned societies and schol¬ 
arly publishers; a factor complicated as some societies now out¬ 
source publishing of their journals to commercial entities. As 
the legitimacy of those institutes is tied to the credibility of the 
work that they publish, the impact of evolving journal-coupled 
peer review systems can have quite different implications for 
their relative standing among research communities. While de¬ 
velopments such as Open Access have clearly catalysed innova¬ 
tions in peer review, it is the whole scholarly ecosystem that is 
evolving in a range of different ways. This has important ram¬ 
ifications for the long-term sustainability of scholarly peer re¬ 
view, and the social aspects that currently govern the different 
practices. 

Perhaps the biggest innovation is that of the increasing trend 
of ‘open peer review’(Parks and Gunashekar 2017), which it¬ 
self has become a quite convoluted term (Ross-Hellauer, Deppe 
and Schmidt 2017; Ross-Hellauer 2017a) within part of broader 
developments in 'open science’. It has been diagnosed to 
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refer to seven key aspects of peer review: open identities, open 
reports, open participation, open interaction, open pre-review 
manuscripts, open final-version commenting and open plat¬ 
forms (or ‘decoupled review’) (Ross-Hellauer 2017a). Journals 
and scholarly publishers are now experimenting with various 
combinations of these traits, in order to find what works best 
in terms of providing verification, reputation/certification and 
incentivisation, while balancing transparency within a peer re¬ 
view culture in which opacity is often regarded as the norm, to 
various degrees (Rooyen et al. 1999; van Rooyen et al. 2010; Parks 
and Gunashekar 2017; Ross-Hellauer, Deppe and Schmidt 2017; 
Allen et al. 2018). 

In spite of a general ecosystem shift towards openness, it is 
perhaps fair to say that those who have been most progressive in 
this regard are the newer 'born open’ publishers, who have the 
distinct advantage of firstly being able to build new communities 
from scratch with different standards, but also not disrupting 
their own traditions and business models. For example, BioMed 
Central, Elife, Frontiers, Copernicus, the Self-Journal of Science, 
PeerJ and F1000 Research represent a range of these ‘born open’ 
publishers (both for-profit and non-profit) who have adopted dif¬ 
ferent and innovative aspects of open peer review since their 
beginnings. Very few publishers or platforms seem to fulfill the 
complete combination of all 7 traits, with exceptions such as Sci- 
enceOpen. 

Perhaps one of the most critical innovations accompanying 
this diversification was that of ‘soundness-only’ peer review, of¬ 
ten considered a defining trait for megajournals, in which only 
the scientific rigour of research, not purported novelty or impact, 
was a deciding factor in publication (Spezi et al. 2017). This prin¬ 
ciple is more closely aligned with the original learned-society 
managed process of peer review. Nonetheless, virtually all of 
these innovations are still centralised around the concept of 
journals and articles. Even ‘publishing platforms’ are essentially 
still journals, functionally equivalent to a megajournal (Ross- 
Hellauer, Schmidt and Kramer 2018), and therefore are only a 
small step towards migrating into a fully Web-literate and net¬ 
worked mode of peer review and publishing. 

Preprints and post-publication peer review 

One of the first platforms launched on the Web was arXiv 
in 1991. In numerous sub-disciplines of the physical sciences, 
mathematics and computer sciences, researchers share non¬ 
peer reviewed manuscripts to arXiv, which currently publishes 
around 100 000 manuscripts each year (known as preprints or 
e-prints) (Ginsparg 2016; Pulverer 2016). Here, the purpose is for 
community-driven cost-effective and rapid communication of 
research results for collaboration and feedback, which has had 
differential uptake across the various research disciplines that 
use arXiv (Marra 2017). Preprints are currently experiencing an 
explosive wave of growth in a variety of disciplines, catalysed 
by a wide range of different tools, platforms and community- 
level organisations (e.g. ASAPbio, PREreview), often targeted at 
specific communities that are already adopting preprint services 
(Tennant et al. 2018). Overlay journals are services that exist by 
leveraging the existing structures of platforms like arXiv, with 
community organised peer review acting as a layer on top of this 
and the ‘journal’ itself being a collection of links to peer reviewed 
preprints. 

With the ongoing disciplinary expansion in preprint servers 
(e.g. biorXiv, multiple servers powered by the Open Science 
Framework), there is an increasing scope for a number of new 
overlay journals to be developed, tailored for different research 


communities. Services such as F1000 Research are similar to 
preprint platforms, where papers are made available prior to 
successive iterations of peer review, with manuscripts updated 
through a simple system of version control. Other services such 
as PubPeer, PaperHive and ScienceOpen provide a range of post¬ 
publication services, typically both on preprints and final ver¬ 
sion manuscripts. 

There remain enormous challenges here in interoperabil¬ 
ity between vendors, formal recognition of the preprint and 
‘post-publication peer review’ process, recognition of the re¬ 
views themselves, which can often remain difficult to discover, 
and then using such reviews to alter published articles, which 
are often considered to be final (and therefore immutable); a 
problem exacerbated by the ubiquitous usage of the PDF for¬ 
mat and lack of version control. Aggregating reviews from across 
platforms, and then formalising their recognition as a method 
of scholarly evaluation is the clear next step here in creating a 
more continuous peer review and publication workflow (Florian 
2012; Kriegeskorte 2012). An interesting consequence of these 
platforms and services is that initial communication is decou¬ 
pled from formal journal-based publishing, and new vendors are 
now increasingly finding ways of integrating peer review into 
preprint platforms. This has incredibly important consequences 
on the wider scholarly publishing industry, who must now find 
ways of justifying their added value, such as journal branding 
and archiving, once the critical processes of dissemination and 
peer review have been decoupled from them. Similarly, there 
is now an increasing responsibility for the research communi¬ 
ties adopting preprint platforms to find ways of developing a 
common infrastructure around preprints, coupled with an ex¬ 
plicit scholarly governance model in which accountability is a 
core trait. Without this, preprints and novel forms of peer review 
around them will never acquire the same level of legitimacy as 
journal-based processes. 


Credit for peer review 

How to provide and receive appropriate credit for peer review is 
an ongoing debate. Recently, Crossref, the primary Digital Ob¬ 
ject Identifier (DOI) provider for scholarly research, announced 
that review reports could be now registered as part of their ser¬ 
vices (Lin 2017). This helps to solve the issues of permanent 
identification and citation of review reports, enabling their wider 
re-use. Other platforms, such as Publons, provide researchers a 
way to keep a track of their review record, and integrate this 
into academic profiles such as ORCID. The focus here is on fa¬ 
cilitating credit for peer review, but not actually providing any 
sort of accreditation themselves—this decision is still based on 
those in charge of research assessment. While Publons provides 
a method of allowing authors and other parties to rate review 
contributions, the primary focus is still on the simple recogni¬ 
tion that a review was performed, rather than the intrinsic qual¬ 
ity and value of that review. ScienceOpen is a discovery engine 
that allows researchers to review both preprints and published 
articles, with each review receiving a CC BY license and Cross¬ 
ref DOI to encourage citation and re-use, and the potential to 
integrate with Publons and ORCID. There is, therefore, currently 
a great potential scope of providing more detailed information 
about peer review quality, in a manner that is further tied to re¬ 
searcher reputation and certification. The main barrier that re¬ 
mains here is the fact that peer review is still largely a closed and 
secretive process, which inhibits the distribution of any form of 
credit. 
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THE FUTURE OF PEER REVIEW 

What would scholarly publishing look like if we rebuilt it from 
scratch using the tools and knowledge available to us in 2018? 
This question is not theoretically or conceptually difficult to ex¬ 
plore. However, it is problematic often to even discuss, due to 
the instantaneous resistance that comes because we are talk¬ 
ing about disruption of an incredibly complex system adopted 
by a powerful and thriving industry, and one in which cultural 
and social norms are deeply embedded across multi-stakeholder 
processes and institutes. Due to the powerful status of peer re¬ 
view in granting a means of academic capital and prestige, it 
has gradually evolved to become part of an increasingly bureau¬ 
cratic and neo-liberal institutional process, which can stifle in¬ 
novation. Nonetheless, it is a powerful thought process to ex¬ 
plore, as essentially it represents a collective vision that most 
stakeholders in scholarly communications have to streamline 
the processes, but with extremely different ideas about the time 
frame that such a vision would be possible to realise in, as well 
as how to achieve it. Coupled with this, serious consideration is 
required into whether or not peer review requires a standard, 
grounded in transparency, in order to be verifiable across a di¬ 
verse range of communities. This would introduce substantially 
more rigour into the process, which we should expect from such 
a critical part of scholarly research. 

One key element of this future is the continued decoupling 
of peer review from journals, through ongoing developments in 
preprints and community-organised peer review, as discussed 
above. There is a potential here that researchers begin to see 
journals as redundant, beyond services such as branding and 
archiving, and therefore we start to see publishers diversify 
and unbundle their publishing services. Such could be achieved 
through the offering or ‘unbundling’ of ‘freemium’ services, such 
as English-language proofing, copy-editing, type-setting, plagia¬ 
rism checks and press and media services. Now, large scholarly 
publishers such as Elsevier are even rebranding as data and ana¬ 
lytics companies, perhaps catalysed by the recognition that jour¬ 
nals will have significantly less value in the future. However, it 
is extremely unlikely that the wider scholarly publishing indus¬ 
try will require, or encourage, such a radical shift into services 
like this, while journal brands are still a dominant factor govern¬ 
ing research assessment processes (Brembs, Button, and Munafo 
2013). This is perhaps best emphasised by the relatively slow 
growth of platforms that offer such ‘decoupled’ services, includ¬ 
ing Peerage of Science and Rubriq, as well as the shutting down 
of Axios Review in early 2017 (Rajagopalan 2017), in comparison 
to an otherwise rapidly growing publishing industry. Therefore, 
the emergence of new services must pay heed to, and where ap¬ 
propriate even influence, wider changes happening in research 
impact, reputation and evaluation, which strongly influence au¬ 
thor choice on publishing venue. This is where the key aspect 
of certification comes in—it is vastly inappropriate for any new 
service to discuss researcher incentives for engaging with new 
models, while not having those incentives formally recognised 
and valued by those in charge of evaluation and career pro¬ 
gression. In order for any aspect of this to achieve progress, 
there must be a thorough critical discourse about the func¬ 
tion of peer review, including knowledge gaps, in order to help 
the different stakeholders to formulate strong evidence-based 
policies. 

In almost every aspect of the Web, different communities are 
embracing the power of networks to evaluate diverse forms of 
information. Scholarly communication is clearly lagging behind 
this, and in the future, we anticipate the more widespread adop¬ 


tion of collaborative technologies that take advantage of such 
social processes. These Web-based technologies have the great 
potential of bridging the presently fragmented landscape of par¬ 
ties interested in peer review (Grimaldo, Marusic and Squazzoni 
2018), helping to resolve the general lack of data sharing (Lee 
and Moher 2017), and providing an accelerated cultural shift to¬ 
wards novel and optimised forms of peer review and research 
evaluation. 

Within different communities and disciplines, there is still 
a great need for solving issues to do with the exclusivity (Flier 

2016) , the anonymity, the time and expense (Copiello 2018), the 
accountability, the subjectivity and bias (Lee et al. 2013), resolv¬ 
ing conflicts of interest (Resnik and Elmore 2018), the recognition 
(Pontille and Torny 2015; Papelis and Petty 2018) and the slow 
publisher-driven nature of the peer review process (Epstein et al. 

2017) . Finding the balance between dissemination and valida¬ 
tion, reconciled between the different stakeholder groups, will 
be a key element of this. However, this incredible dimension¬ 
ality of difficulties should indicate to us that the problems with 
modern peer review are systemic and encourage us to think out¬ 
side of the black box of the journal-coupled process to what any 
modern suite of functions should look like. 

As an example of this, Tables 1-3 emphasis the potential 
different solutions that a hypothetical fully collaborative, Web- 
enabled process of peer-to-peer review would bring to the many 
of the issues currently levied at peer review (Priem and Hem- 
minger 2012; Kovanis et al. 2017; Tennant et al. 2017). These are 
provided in the critical contexts of quality control and moder¬ 
ation (Table 1), certification and reputation (Table 2) and incen¬ 
tives for engagement (Table 3). Only by harmonising all three of 
these will any successful and sustainable model of peer review 
be enabled. By illustrating the distinction in this way, it is em¬ 
inently feasible for any existing or new platform to adopt just 
one or several of the proposals, rather than a full-scale transfor¬ 
mation of the present system. What this represents is a con¬ 
ceptual vision of what is possible, based on existing services, 
and therefore it is eminently possible for individual factors to be 
taken up by the present journal-based system. However, as they 
are all based on traits from existing services (e.g. from GitHub, 
Wikipedia, or Stack Exchange), it would also be quite possi¬ 
ble for them to by all modelled as a single, hybrid construct, if 
desired. 

In Table 1, the critical aspect that would define success would 
be the uptake of any open participation model, such that it 
was seen as a genuine alternative, not an add on, to formal 
methods of peer review. These openly collaborative models are 
already proving highly successful where available, such as with 
the range of journals published by Copernicus on behalf of the 
European Geosciences Union (Poschl and Koop 2008; Poschl, 
2010, 2012). Therefore, there is little stopping any of these in¬ 
dividual traits becoming adopted by the present journal-based 
system, and they could have governance structures maintained 
by learned societies. This would provide a strong way of shifting 
towards a fairer and more community-managed processes, as 
well as embedding additional transparency, accountability, and 
legitimacy into ‘editorial’ processes. Providing this solution in 
a sustainable manner across disciplines would require a wider 
change in culture, based on the recognition that such processes, 
despite being coupled to journals, have proven to be highly suc¬ 
cessful in the Geosciences. Other Open Access publishers, such 
as Frontiers and eLIFE, which also practice forms of collabora¬ 
tive peer review, will be highly important here in demonstrat¬ 
ing that open participation can work well in other disciplines. 
In order to increase the adoption of this, it will be necessary for 
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Table 1. Potential future for quality control and moderation. 

Traditional 


Future 

Gatekeeping function as a selective content filter 

Quality control difficult to measure, with little real 
evidence of success 

Secretive and selective review within a closed system 
Organised around journals and papers 

Non-accountable due to ‘black box’ of editorially-controlled 
process 

Structurally limited and exclusive, usually to 2-3 people 
Legitimacy conferred by reputation of brands and editors 


No gatekeeping, collaboration and constructive criticism define filters 
Quality control achieved based on consensus, with evaluation based 
on engagement 

Self-organised, open and unrestricted communities 

Unrestricted content types and formats 

Elected moderators accountable to their respective communities 

Open participation, with semi-automated review matching 

Legitimacy provided as a community governed process 

Table 2. Potential future for certification and reputation. 

Traditional 


Future 

Poorly recognised and rewarded activity for researchers 

Difficult to measure due to the opacity of the process 

Often defaulted to inappropriate higher-level proxies 

Closed process of identification prohibits recognition 

High reviewer turn-down rates, and general frustration for all parties 
Level of entry high, based on editorial decision and knowledge 

Little incentive for those in charge of assessments to care 

Performance metrics based on nature and quality of engagement 
Open, continuous community-based evaluation tied to reputation 
Granular, revealed at the object and individual levels 

Fully transparent by default, tied to academic profiles, and portable 
Expanded reviewer pool with greatly reduced barriers to entry 
Engagement filters based on reputation within community 
Appealing for those in charge of assessment due to simplicity 

Table 3. Potential future incentives for engagement. 

Traditional 

Future 


Shared sense of duty, as a natural altruistic incentive 
Researchers generally feel they receive insufficient credit 
Existing incentives only for engagement, with no focus on 
quality 

Incentives decoupled from academic reputation or career 
progression 

Prestige captured by journals to help define their brands 

Same, but with virtual rewards such as points, badges or abilities 

Creates an ‘incentive loop’ to encourage maximum engagement 
‘Reviewing the reviewers’ encourages higher quality engagement 

Coupled to academic records and profiles, and to career advancement 

Establishment of individual prestige as a social process defined by communities 


those publishers to share data on the relative quality of their 
processes compared with traditional peer review methods in or¬ 
der to demonstrate that it is relatively more effective (or not). 

It is impossible to view the potential future model suggested 
in Table 2 decoupled from the incentives outlined in Table 3, as 
there is a strong association between researcher reputation and 
incentives to engage with new processes. This issue is an in¬ 
herently socio-technical one, and one with which the academic 
community has been grappling with as part of its culture for 
some time (Zuckerman and Merton 1971). It is confounded by 
further problems surrounding values privilege, and bias within 
scholarly communication and academic cultures. One of the key 
points here is how to break the association between scholarly 
journals, arguably a 17 th Century mode of communication, and 
the prestige granted to individuals for publishing in them as a 
means of academic career progression. So far, this issue has not 
been concretely resolved, despite decades of understanding the 
issues associated with it, and numerous alternative proposals. 
Campaigns, such as the San Francisco Declaration on Research 
Assessment (DORA), that call for great rigour and transparency 
in research assessment, do not seem to have had any significant 
impact on researcher behaviours; if they had, we would have ex¬ 
pected to see a weakening of journals as the primary mode of 


scholarly communication, which has not occurred. Indeed, it is 
likely that this academic perception of journals as the authori¬ 
tative source for research, in part due to the apparent verifica¬ 
tion and certification role that peer review plays when coupled 
to it, has stifled much of the innovation beyond journal-based 
peer review in many disciplinary communities (Nicholas et al. 
2015). Therefore, one key element to improve this state is that of 
providing sufficient training and support, particularly for more 
inexperienced or at-risk reviewers, as well as risk-mitigation 
strategies, that would enable researchers to be comfortable ex¬ 
perimenting with new forms of peer review and scholarly com¬ 
munication. 

The key element in Table 3 for incentives is the attempt to 
capture and define different levels of researcher prestige. At the 
present, the prestige or reputation of an individual, or individ¬ 
ual piece of work, is often tied with journal brands by proxy, but 
is also an incredibly multi-dimensional concept to comprehend 
or measure; for example, institutional status, intrinsic biases 
and privilege and community values and norms. It is difficult 
to simplify or change this, due to the coupling of prestige with 
career advancement (Moore et al. 2017); therefore, the key will be 
demonstrating not that any new method of recognition not only 
out-performs present models (Kovanis et al. 2017), but that they 
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do so by providing an enriched insight into researcher prestige 
in a complimentary manner to traditional methods. For exam¬ 
ple, expanding what it is possible to obtain credit for to include a 
more diverse suite of research outputs (e.g. data, code and soft¬ 
ware, images, instructional videos) and coupling this with how 
that content is digested and engaged with by the wider commu¬ 
nity should be of considerable interest to those who wish to pro¬ 
vide a fairer and more rigorous process of research evaluation, 
and, in particular, learned societies. 

As such, this is why tying additional forms of academic en¬ 
gagement, such as peer review, teaching and public outreach, 
with certification and reputation (Table 2) will be a critical as¬ 
pect to consider for any future innovations in this field. This, in 
turn, relies on getting buy-in from those who are in charge of re¬ 
search assessment, including research funders and hiring com¬ 
mittees, which will be pivotal in defining more holistic forms of 
reputation attainment in order to incentivise more diverse forms 
of research activity. Indeed, it is likely that a systemic failure to 
convince institutes as to the value of peer review for academic 
capital, combined with industrial inertia, has been one of the 
strongest barriers towards providing sufficient incentives for in¬ 
novations in peer review. However, with the growth of compa¬ 
nies like Publons that seek to provide credit for referees, and 
their recent acquisition by Clarivate Analytics, we might be en¬ 
couraged that such reputational incentives might become more 
firstly increasingly measurable, and secondly more institution¬ 
ally embedded. In the future, we might expect to see similar 
initiatives being designed by scholarly communities under their 
own control, in which they are able to define and regulate cer¬ 
tification and accreditation protocols. There is a great potential 
here to leverage either centralised or decentralised peer-to-peer 
networks to guide recognition and evaluation in scholarly com¬ 
munication (Hartgerink and van Zelst 2018). 


CONCLUSIONS 

The conceptual framework which is outlined here is generally 
concordant with broader changes in the 'open science’ move¬ 
ment, reflecting needs for greater transparency in research pro¬ 
cesses and outputs. While peer review is now an almost ex¬ 
clusively Web-based process now, much of it, and those who 
adopt it, are still based on non-digital communication norms. 
The framework outlined here was designed in mind to stimulate 
further discussion into this issue, and to help increase the reli¬ 
ability of peer review while accounting for some of the caveats 
associated with innovations in peer review. It also has the po¬ 
tential to help shape a more rigorous method of scholarly evalu¬ 
ation and assessment that could help to simultaneously resolve 
issues to do with traditional journal-based methods of commu¬ 
nication and ranking, something that is critically required for 
the modern academy (Brembs, Button, and Munafo 2013; Moore 
et al. 2017; Brembs 2018). The proposal is embedded in prin¬ 
ciples of open scholarly communication, including inclusivity 
and open engagement, which are distinct from the tradition¬ 
ally closed and exclusive models of journal-coupled peer review. 
There is little preventing such changes being adopted as part 
of a strategic stepwise change within the present publishing in¬ 
dustry, to allow for the reformation and adaptation of existing 
systems, evidence gathering and cultural behaviour to evolve. 

All of this potential for innovation in peer review demands 
that we continue to ask serious questions about the present 
scholarly communication ecosystem. For example, what are the 
roles of editors, librarians, and publishers in any proposed or 


hypothetical future system? What will the impact of any such 
innovation be on different communities with different social 
norms, research practices and inherent biases? How do we re¬ 
solve the tensions between actors who want rapid transforma¬ 
tion of peer review, and those who are more conservative or en¬ 
trenched within the present status quo? 

These are not easy questions, and there are certainly not 
any easy answers. In spite of this, we would like to see con¬ 
tinued critical discussion on many of these elements, as well 
as a removal of the fear to innovate, acknowledgement of any 
weaknesses, recognition of layers of accountability and the de¬ 
sire to embrace a more diverse thought process around peer re¬ 
view and scholarly communication; all while minimising risk to 
those who wish to innovate, and making sure that the present 
power dynamics within scholarly communication are not sim¬ 
ply recapitulated in any new system. The key question that 
unifies the above is why there seems to have been such a low 
uptake of the different innovative aspects of peer review, when 
features such as decoupled review, credit enabling and open 
participation have been around in different forms now for some 
time. It is likely that there are three primary answers to this, 
involving a general lack of evidence into the peer review pro¬ 
cess at different scales, the apparent decoupling of peer review 
from any sort of formalised recognition for academic career ad¬ 
vancement and the above-mentioned perception of risk asso¬ 
ciated with non-traditional processes of scholarly communica¬ 
tion. Therefore, these are the barriers that will likely require 
most attention in the future of peer review and scholarly com¬ 
munication innovation, and learned societies are perhaps best 
placed to lead this with the support of their respective commu¬ 
nities (Prechelt, Graziotin, and Fernandez 2018). 

In spite of this, there does however appear to be an emerg¬ 
ing wave of momentum and support for disrupting peer re¬ 
view, largely fuelled by social organisations such as ASAPbio, 
which aims to increase transparency and innovation in the Life 
Sciences in particular (http://asapbio.org/). This has coincided 
with a developing understanding of peer review, thanks to the 
work of initiatives such as PEERE (http://www.peere.org/). The 
key to maintaining this momentum will be sustained engage¬ 
ment with the different stakeholders to develop a more holis¬ 
tic framework of peer review, in which risk perception is min¬ 
imised while the advantages are made much more explicit and 
evidence-based (Rennie 2016). 

We anticipate that future discussions and innovations will 
focus on a number of particular areas: 

* The question of sustainability in peer review, what this 
means for the different actors involved in the process, and 
how to demonstrate that innovative models are superior to 
existing ones; 

• How to catalyse wider participation in the discussions and 
innovations in peer review, bearing in mind the incredible so¬ 
cial, cultural and practical diversity across disciplines; 

* The impact of developments in peer review in different com¬ 
munities, including dimensions of bias and potential socio- 
technological innovations required to overcome this; 

• Whether or not innovations reinforce or disrupt entrenched 
norms between different research communities; 

• A critical appraisal of how to create a more diverse and equi¬ 
table future for peer review, including the role of peer review 
in research evaluation processes; 

* The role of traditional forms of communication (i.e. journals) 
and non-community owned publishing platforms, particu¬ 
larly with respect to governance structures; 
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• How to close the divergence between the original ideal of 
peer review (and whether this needs to be critically ap¬ 
praised) and the modern practice of it; 

• And finally, how Internet-style communication norms can be 
integrated into peer review, and why our expectations for this 
to happen seem to be lagging for scholarly publishing and 
peer review. 

While we should not encourage conformation to the status 
quo in scholarly communication, and a general lack of experi¬ 
mentation, we should also be fully sympathetic towards stake¬ 
holders who might not want to see such disruption of scholarly 
communication norms. Thus, engagement efforts should be fo¬ 
cused more on understanding what the reasons for this might 
be and to use this knowledge to see how to bring what is best for 
different communities into line with that. There appears to be a 
general apathetic view towards many aspects of scholarly com¬ 
munication, and it is the responsibility of those who are help¬ 
ing to sculpt this future to maximise participation in it through 
effective communications. Then, the global scholarly commu¬ 
nity can collectively help to address the real issues of control 
and governance of public research. It is our hope that this paper 
highlights the incredible scope for potential innovations in the 
future of peer review, and that different communities draw in¬ 
spiration from that to design optimal systems of research com¬ 
munication. 
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